38

Algorithms for Binary Neural Networks

and that real-valued neurons may not even be required in deep neural networks, which is

comparable to the idea behind biological neural networks.

Additionally, an efficient method to examine the interpretability of deep neural networks

is to reduce the bit width of a particular layer and examine its impact on accuracy. Numerous

works [199, 159] investigate how sensitive various layers are to binarization. In common

BNNs, the first and last layers should, by default, be kept at higher precision. This means

that these layers are more crucial for predicting neural networks. This section attempts to

state the nature of binary neural networks by introducing some representative work.

3.2

BNN: Binary Neural Network

Given an N-layer CNN model, we denote its weight set as W = {wⁿ}^N

n=1 ^{and the input}

feature map set as A = {aⁿ

in^}N

n=1^{. The wn ∈RCn}

out^×Cn

in^×Kn×Kn and aⁿ

in ^∈RCn

in^{×W n}

in^×Hn

in

are the convolutional weight and the input feature map in the n-th layer, where Cⁿ

in^{, Cn}

out

and Kⁿ, respectively, represent the input channel number, the output channel number, and

the kernel size. In addition, W ⁿ

in ^{and Hn}

in ^{are the width and height of the feature maps.}

Then, the convolutional outputs aⁿ

out ^{can be technically formulated as:}

aⁿ

out ^{= wn ⊗an}

in^,

(3.1)

where ⊗represents the convolution operation. In this book, we omit the non-linear function

for simplicity. Following the prior works [48, 99], BNN intends to represent wⁿ and aⁿ in a

binary discrete set as:

B := {−1(0), +1}.

Thus, the 1-bit format of wⁿ and aⁿ is respectively b^wn ∈B^Cn

out^×Cn

in^×Kn×Kn and b^an

in ∈

B^Cn

in^{×W n}

in^×Hn

in such that the efficient XNOR and Bit-count instructions can approximate

the floating-point convolutional outputs as:

aⁿ

out ^{≈bwn ⊙ban}

in,

(3.2)

where ◦represents channel-wise multiplication and ⊙denotes XNOR and Bit-count instruc-

tions.

However, this quantization mode will cause the output amplitude to increase dramati-

cally, different from the full precision convolution calculation, and cause the homogenization

of characteristics [199]. Several novel objects are proposed to address this issue, which will

be introduced in the following.

3.3

XNOR-Net: Imagenet Classification Using Binary

Convolutional Neural Networks

The scaling factor was first proposed by XNOR-Net [199] to solve this problem. The weights

and the inputs to the convolutional and fully connected layers in XNOR-Nets are approxi-

mated with binary values B.